Infrared imaging device, image processing method, and image processing program

ABSTRACT

An infrared imaging device includes an imaging element including a plurality of infrared detection pixels which are two-dimensionally arranged, a diaphragm, a temperature detection unit that detects the temperature of the diaphragm, and a digital signal processing unit that subtracts, from at least a portion of each of a plurality of captured image data items obtained by capturing images using the imaging element in a state in which an F-number of the diaphragm is set to a plurality of values, a signal value corresponding to an amount of infrared rays which are radiated from the diaphragm and are based on the F-number when each of the plurality of captured image data items is acquired and the temperature detected by the temperature detection unit and combines the plurality of captured image data items after the subtraction to generate composite image data.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation of International Application No.PCT/JP2015/063799 filed on May 13, 2015, and claims priority fromJapanese Patent Application No. 2014-200076 filed on Sep. 30, 2014, theentire disclosures of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an infrared imaging device, an imageprocessing method, and a computer readable medium storing an imageprocessing program.

2. Description of the Related Art

An infrared imaging device, such as a thermographic device or a nightvision device, which captures infrared images has been known (forexample, see JP2013-080130A, JP2008-278036A, and JP1997-101207A(JP-H09-101207A)).

JP2013-080130A discloses an infrared imaging device in which an imagingoptical system includes a diaphragm.

JP2008-278036A discloses an infrared imaging device which minimizes theopening area of a diaphragm included in an imaging optical system toprevent the damage of an imaging element in a case in which an image ofa high-temperature object is captured.

JP1997-101207A (JP-H09-101207A) discloses an infrared imaging devicewhich controls a diaphragm according to the highest temperature amongthe temperatures of an object, of which the image is being captured, tochange a temperature measurement range.

SUMMARY OF THE INVENTION

The infrared imaging device disclosed in JP2008-278036A controls theopening area of the diaphragm so as to be minimized in a case in whichthe image of a high-temperature object is captured. Therefore, theamount of infrared rays which reach the imaging element from alow-temperature object is reduced and it is difficult to accuratelycapture the image of the low-temperature object.

The infrared imaging device disclosed in JP1997-101207A (JP-H09-101207A)changes the measurement temperature range depending on the highesttemperature of the object. Therefore, in a case in which the highesttemperature is significantly high, sensitivity to infrared rays from alow-temperature object is reduced and it is difficult to accuratelycapture the image of the low-temperature object.

JP2013-080130A does not disclose a technique that controls the openingarea of the diaphragm. Therefore, it is difficult to improve imagingquality in a scene in which both a low-temperature object and ahigh-temperature object are present.

The invention has been made in view of the above-mentioned problems andan object of the invention is to provide an infrared imaging device, animage processing method, and a computer readable medium storing an imageprocessing program that can improve imaging quality in a scene in whichboth a low-temperature object and a high-temperature object are present.

An infrared imaging device according to the invention comprises animaging element including a plurality of infrared detection pixels whichare two-dimensionally arranged, a diaphragm that is provided closer toan object than the imaging element, a temperature detection unit thatdetects a temperature of the diaphragm, and an image processing unitthat subtracts, from at least a portion of each of a plurality ofcaptured image data items obtained by capturing images using the imagingelement in a state in which an F-number of the diaphragm is set to aplurality of values, a signal value corresponding to an amount ofinfrared rays which are radiated from the diaphragm and are based on theF-number when each of the plurality of captured image data items isacquired and the temperature detected by the temperature detection unitand combines the plurality of captured image data items after thesubtraction to generate composite image data.

An image processing method according to the invention is performed by aninfrared imaging device including an imaging element including aplurality of infrared detection pixels which are two-dimensionallyarranged, a diaphragm that is provided closer to an object than theimaging element, and a temperature detection unit that detects atemperature of the diaphragm. The image processing method comprises animage processing step of subtracting, from at least a portion of each ofa plurality of captured image data items obtained by capturing imagesusing the imaging element in a state in which an F-number of thediaphragm is set to a plurality of values, a signal value correspondingto an amount of infrared rays which are radiated from the diaphragm andare based on the F-number when each of the plurality of captured imagedata items is acquired and the temperature detected by the temperaturedetection unit and combining the plurality of captured image data itemsafter the subtraction to generate composite image data.

An image processing program according to the invention causes aninfrared imaging device including an imaging element including aplurality of infrared detection pixels which are two-dimensionallyarranged, a diaphragm that is provided closer to an object than theimaging element, and a temperature detection unit that detects atemperature of the diaphragm to perform an image processing step ofsubtracting, from at least a portion of each of a plurality of capturedimage data items obtained by capturing images using the imaging elementin a state in which an F-number of the diaphragm is set to a pluralityof values, a signal value corresponding to an amount of infrared rayswhich are radiated from the diaphragm and are based on the F-number wheneach of the plurality of captured image data items is acquired and thetemperature detected by the temperature detection unit and combining theplurality of captured image data items after the subtraction to generatecomposite image data.

According to the invention, it is possible to provide an infraredimaging device, an image processing method, and an image processingprogram that can improve imaging quality in a scene in which both alow-temperature object and a high-temperature object are present.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically illustrating the structure of aninfrared imaging device for describing an embodiment of the invention.

FIG. 2 is a diagram illustrating an FPN calculation method using twocaptured image data items obtained by capturing image while the F-numberis changed to F1 and F1.4.

FIG. 3 is a diagram illustrating the relationship between the F-numberand infrared transmittance α indicating the amount of infrared raystransmitted through a diaphragm 2.

FIG. 4 is a flowchart illustrating an FPN calculation operation of theinfrared imaging device illustrated in FIG. 1.

FIGS. 5A, 5B, and 5C are diagrams illustrating the diaphragm 2 as viewedfrom an imaging element 3.

FIG. 6 is a diagram illustrating the relationship between the F-numberand the amount of infrared rays radiated from the diaphragm 2.

FIG. 7 is a diagram illustrating a modification example of the FPNcalculation method.

FIG. 8 is a diagram illustrating the detection sensitivity of eachinfrared detection pixel when the imaging element 3 captures an image ofan object with a uniform temperature.

FIG. 9 is a diagram illustrating a pixel signal value output from aninfrared detection pixel at a position “L” or “R”.

FIG. 10 is a diagram illustrating a change in the pixel signal valueoutput from each infrared detection pixel of the imaging element 3 withrespect to the temperature of the object (output responsecharacteristics of the infrared detection pixel).

FIG. 11 is a diagram illustrating a calculation coefficient a.

FIG. 12 is a diagram illustrating a calculation coefficient b.

FIG. 13 is a diagram illustrating the combination result of two capturedimage data items.

FIG. 14 is a diagram illustrating an example of the object of which theimage is captured by the infrared imaging device illustrated in FIG. 1.

FIGS. 15A and 15B are diagrams illustrating captured image data obtainedby capturing the image of the object illustrated in FIG. 14 while theF-number of the diaphragm 2 is changed to two values.

FIG. 16 is a diagram illustrating composite image data 155 obtained bycombining captured image data 151 and captured image data 152.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the invention will be described withreference to the drawings.

FIG. 1 is a diagram schematically illustrating the structure of aninfrared imaging device for describing an embodiment of the invention.

The infrared imaging device illustrated in FIG. 1 comprises an imaginglens 1, a diaphragm 2, an imaging element 3 that captures an image of anobject through the imaging lens 1 and the diaphragm 2, a temperaturedetection unit 4 that detects the temperature of the diaphragm 2, ananalog signal processing unit 6, an analog/digital conversion circuit(A/D conversion circuit) 7, a diaphragm driving unit 9, and an imagingelement driving unit 10.

The imaging element 3 includes a plurality of infrared detection pixelswhich are two-dimensionally arranged and detect infrared rays (ingeneral, light with a wavelength of 8 μm to 12 μm) that have beenradiated from the object and focused through the imaging lens 1 and thediaphragm 2. The imaging element 3 is driven by the imaging elementdriving unit 10.

A pyroelectric element is given as an example of an infrared detectionelement used in the infrared detection pixel. Alternatively, athermopile-type infrared detection element in which thermocouplesgenerating the Seebeck effect are connected or a bolometer-type infrareddetection element using a change in a resistance value caused by atemperature rise may be used.

The infrared detection element is not limited to the above and any typeof infrared detection element may be used as long as it can detectinfrared rays. In the specification, a set of pixel signals output fromall of the infrared detection pixels of the imaging element 3 isreferred to as captured image data.

The diaphragm 2 is provided closer to the object than the imagingelement 3 and the opening area of the diaphragm 2 is controlled by thediaphragm driving unit 9.

The temperature detection unit 4 is a temperature sensor, such as athermistor, is provided in the vicinity of the diaphragm 2, and notifiesa system control unit 11 of the detected temperature.

The analog signal processing unit 6 performs analog signal processingincluding an amplification process of multiplying each pixel signalvalue of the captured image data output from the imaging element 3 bythe gain to amplify the captured image data.

The A/D conversion circuit 7 converts the analog captured image dataoutput from the analog signal processing unit 6 into digital data.

The analog signal processing unit 6, the A/D conversion circuit 7, thediaphragm driving unit 9, and the imaging element driving unit 10 arecontrolled by the system control unit 11.

A command signal is input from the user to the system control unit 11through an operating unit 14.

In addition, an electric control system of the infrared imaging devicecomprises a main memory 16, a memory control unit 15 connected to themain memory 16, a digital signal processing unit 17 that performs, forexample, a process of converting the captured image data output from theA/D conversion circuit 7 into a format which can be displayed by adisplay unit 23, an FPN calculation unit 19 that calculates FPN includedin the captured image data obtained by capturing an image using theimaging element 3, an external memory control unit 20 to which adetachable recording medium 21 is connected, and a display control unit22 to which the display unit 23 is connected.

The memory control unit 15, the digital signal processing unit 17, theFPN calculation unit 19, the external memory control unit 20, and thedisplay control unit 22 are connected to each other by a control bus 24and a data bus 25 and are controlled by instructions from the systemcontrol unit 11.

The FPN calculation unit 19 calculates FPN included in the capturedimage data obtained by capturing the image of the object using theimaging element 3, on the basis of first captured image data obtained bycapturing the image of the object using the imaging element 3 in a statein which the F-number of the diaphragm 2 is set to a first value, secondcaptured image data obtained by capturing the image of the object usingthe imaging element 3 in a state in which the F-number of the diaphragm2 is set to a second value, the first value, and the second value.

FIG. 2 is a diagram illustrating an FPN calculation method in a case inwhich the first value is F1 and the second value is F1.4.

In FIG. 2, reference numeral 31 indicates a pixel signal value at anarbitrary coordinate position in the first captured image data obtainedin a state in which the F-number=F1 is established. Reference numeral 32indicates a pixel signal value at the arbitrary coordinate position inthe second captured image data obtained in a state in which theF-number=F1.4 is established.

As illustrated in FIG. 2, each pixel signal value 31 of the firstcaptured image data and each pixel signal value 32 of the secondcaptured image data include an FPN component C_(fpn) and a signalcomponent SG other than the FPN component C_(fpn).

Here, the first captured image data and the second captured image dataare acquired at a short time interval (for example, the first capturedimage data and the second captured image data are continuouslycaptured). Therefore, the FPN components C_(fpn) included in each pixelsignal value 31 of the first captured image data and each pixel signalvalue 32 of the second captured image data can be considered to have thesame value.

FIG. 3 is a diagram illustrating the relationship between the F-numberand infrared transmittance α indicating the amount of infrared raystransmitted through the diaphragm 2. In FIG. 3, the infraredtransmittance α through the diaphragm 2 when the F-number=F1 isestablished (a state in which the opening area of the diaphragm 2 is themaximum) is standardized to “100%”.

As illustrated in FIG. 3, when the F-number=F1.4 is established, theamount of infrared rays incident on the imaging element 3 is half of theamount of infrared rays incident on the imaging element 3 when theF-number=F1 is established.

That is, a signal component SG (hereinafter, also represented by SG(2))included in each pixel signal value 32 of the second captured image dataillustrated in FIG. 2 is a value obtained by multiplying a signalcomponent SG (hereinafter, also represented by SG(1)) included in eachpixel signal value 31 of the first captured image data by the ratio(=0.5) of the infrared transmittance α at an F-number of F1.4 to theinfrared transmittance α at an F-number of F1.

As such, the ratio of the signal component SG(1) included in the pixelsignal value 31 to the signal component SG(2) included in the pixelsignal value 32 is uniquely determined from the difference between theF-number when the pixel signal value 31 is obtained and the F-numberwhen the pixel signal value 32 is obtained.

The FPN calculation unit 19 sets a coefficient for making the value ofthe signal component SG(2) and the value of the signal component SG(1)equal to each other, on the basis of the ratio (=0.5) of the infraredtransmittance α at an F-number of F1.4 to the infrared transmittance αat an F-number of F1, and multiplies at least one of the pixel signalvalue 31 or the pixel signal value 32 by the set coefficient.

For example, in a case in which only the pixel signal value 31 ismultiplied by the coefficient, the ratio (=0.5) of the infraredtransmittance α at an F-number of F1.4 to the infrared transmittance αat an F-number of F1 is set as the coefficient.

In a case in which only the pixel signal value 32 is multiplied by thecoefficient, the reciprocal (=2) of the ratio (=0.5) of the infraredtransmittance α at an F-number of F1.4 to the infrared transmittance αat an F-number of F1 is set as the coefficient.

In a case in which each of the pixel signal value 31 and the pixelsignal value 32 is multiplied by the coefficients, the coefficients areset such that the ratio of the coefficients by which each pixel signalvalue is multiplied is equal to the ratio (=0.5) of the infraredtransmittance α at an F-number of F1.4 to the infrared transmittance αat an F-number of F1.

For example, the coefficient by which the pixel signal value 31 ismultiplied is set to “1.5” and the coefficient by which the pixel signalvalue 32 is multiplied is set to “3”.

FIG. 2 illustrates a pixel signal value 33 obtained by multiplying thepixel signal value 32 by a coefficient (=2). The multiplication of thecoefficients makes the signal component SG of the pixel signal value 31and the signal component SG of the pixel signal value 33 have the samevalue.

Therefore, after multiplying the coefficients, the FPN calculation unit19 can subtract the pixel signal value 31 from the pixel signal value 33to calculate the FPN component C.

In a case in which only the pixel signal value 31 is multiplied by thecoefficient, a pixel signal value obtained by multiplying the pixelsignal value 31 by 0.5 can be subtracted from the pixel signal value 32and a pixel signal value after the subtraction can be multiplied by 2 tocalculate the FPN component C.

In a case in which each of the pixel signal value 31 and the pixelsignal value 32 is multiplied by the coefficients, a pixel signal valueobtained by multiplying the pixel signal value 31 by 1.5 can besubtracted from a pixel signal value obtained by multiplying the pixelsignal value 32 by 3 and a pixel signal value after the subtraction canbe divided by 1.5 to calculate the FPN component C_(fpn).

FIG. 4 is a flowchart illustrating the FPN calculation operation of theinfrared imaging device illustrated in FIG. 1.

For example, the process illustrated in FIG. 4 is performed at apredetermined interval (for example, once every ten seconds) in a videoimaging mode in which infrared images obtained by capturing the object aplurality of times using the imaging element 3 are displayed on thedisplay unit 23.

In the video imaging mode, it is assumed that the F-number of thediaphragm 2 is set to the first value in advance by, for example, amanual operation when the capture of a motion picture starts.

First, when the capture of a motion picture starts, the digital signalprocessing unit 17 processes the captured image data which is outputfrom the imaging element 3 at a predetermined interval to generaterecording data. The recording data is recorded in the recording medium21 and an infrared image based on the recording data is displayed on thedisplay unit 23.

The processes which are performed for the captured image data by thedigital signal processing unit 17 include a process of subtracting theFPN component C_(fpn) recorded in an FPN recording region of the mainmemory 16 from the captured image data to correct FPN.

When the capture of a motion picture starts, the FPN component C_(fpn)is not recorded in the FPN recording region of the main memory 16.Therefore, when the infrared imaging device is set to the video imagingmode, first, the infrared imaging device performs the following process.

The system control unit 11 maintains the F-number of the diaphragm 2 atthe first value which is set as imaging conditions. In this state, thesystem control unit 11 directs the imaging element 3 to capture an image(Step S1). The first captured image data obtained by the imagingoperation is output from the imaging element 3 and is temporarily storedin the main memory 16 (Step S2).

Then, the system control unit 11 changes the F-number of the diaphragm 2from the first value to the second value. In this state, the systemcontrol unit 11 directs the imaging element 3 to capture an image (StepS3). The second captured image data obtained by the imaging operation isoutput from the imaging element 3 and is temporarily stored in the mainmemory 16 (Step S4).

Then, the FPN calculation unit 19 acquires the first captured image dataand the second captured image data temporarily stored in the main memory16 and calculates the FPN component C_(fpn) on the basis of the acquiredfirst captured image data, the acquired second captured image data, thefirst value which is the F-number when the first captured image data isacquired, and the second value which is the F-number when the secondcaptured image data is acquired (Step S5).

When calculating the FPN component C_(fpn), the FPN calculation unit 19records the FPN component C_(fpn) in the FPN recording region of themain memory 16.

Then, the digital signal processing unit 17 subtracts the FPN componentC_(fpn) generated in Step S5 from each of the first captured image dataand the second captured image data temporarily stored in the main memory16 to correct FPN (Step S6).

Then, the digital signal processing unit 17 performs a process ofmultiplying the second captured image data having the corrected FPN bythe gain such that the difference between the brightness of the secondcaptured image data having the corrected FPN and the brightness of thefirst captured image data having the corrected FPN is reduced (Step S7).

Specifically, the digital signal processing unit 17 multiplies thesecond captured image data having the corrected FPN by the gaincorresponding to the difference between the second value and the firstvalue. The digital signal processing unit 17 functions as a gainprocessing unit.

For example, when the first value is F1 and the second value is F1.4,the gain is 2 on the basis of the difference between the infraredtransmittances a at F1 and F1.4 illustrated in FIG. 2 and the digitalsignal processing unit 17 multiplies each pixel signal value of thesecond captured image data by 2.

When the first value is F2 and the second value is F1.4, the gain is 0.5on the basis of the difference between the infrared transmittances a atF2 and F1.4 illustrated in FIG. 2 and the digital signal processing unit17 multiplies each pixel signal value of the second captured image databy 0.5.

After Step S7, the digital signal processing unit 17 generates recordingdata from the first captured image data and the second captured imagedata processed in Steps S6 and S7 and records the recording data in therecording medium 21 (Step S8).

After Step S8, the system control unit 11 returns the F-number from thesecond value to the first value and continues to capture motionpictures.

As described above, according to the infrared imaging device illustratedin FIG. 1, it is possible to calculate FPN with high accuracy, withoutinterrupting the capture of motion pictures. In addition, a blurredimage is not periodically generated while a motion picture is beingcaptured, unlike the related art. Therefore, it is possible to capturehigh-quality motion pictures.

Furthermore, FPN can be calculated only by an operation ofinstantaneously closing or opening the diaphragm 2 while a motionpicture is being captured, without closing a shutter or withoutgenerating a defocused state. Therefore, it is possible to reduce thepower required to calculate FPN.

According to the infrared imaging device illustrated in FIG. 1, sincethe brightness of the second captured image data is corrected by thegain, the brightness of the recording data which is recorded in therecording medium while a motion picture is being captured is uniform. Inaddition, the brightness of the image which is displayed on the displayunit 23 while a motion picture is being captured is uniform. Therefore,there is no variation in image quality due to a change in the F-numberof the diaphragm 2 while a motion picture is being captured and it ispossible to maintain the quality of a motion picture.

When the F-number of the diaphragm is changed while a motion picture isbeing captured, the depth of field is likely to vary. In an imagingdevice that is mainly used to see an image and captures a visible lightimage, image quality varies significantly with a change in the depth offield. As a result, the user who sees a motion picture feels discomfort.

However, in the infrared imaging device, for example, in a case in whichan image of an object in the plane is captured, a variation in imagequality caused by a change in the depth of field can be allowed.

As such, in the infrared imaging device, there is a range in which achange in the depth of field can be allowed while a motion picture isbeing captured. Therefore, the above-mentioned FPN calculation method iseffective in the infrared imaging device.

The above-mentioned FNP calculation method calculates FPN on the basisof two captured image data items which are acquired while the F-numberis changed. Therefore, when there is a difference in content between thetwo captured image data items, it is difficult to accurately calculateFPN.

For this reason, after Step S4 in FIG. 4, the FPN calculation unit 19compares the first captured image data with the second captured imagedata and determines whether a moving body portion is present in the twocaptured image data items, using a known moving body detection process.The FPN calculation unit 19 functions as a moving body determinationunit.

In a case in which it is determined that a moving body portion ispresent, the FPN calculation unit 19 determines that the accuracy ofcalculating FPN is low and does not perform the process in Step S5.

Then, in Step S6, the digital signal processing unit 17 performs FPNcorrection for the first captured image data and the second capturedimage data, using the FPN stored in the main memory 16 and proceeds toStep S7.

In this way, it is possible to prevent the FPN stored in the main memory16 from being updated with a low-accuracy value and to improve thequality of a motion picture.

The FPN calculation unit 19 may perform the FPN calculation process whena variation in the temperature detected by the temperature detectionunit 4 is equal to or greater than a threshold value. Since a largevariation in the temperature is considered as a large change in FPN,this timing is effective.

In a case in which the FPN calculated by the FPN calculation unit 19 isequal to or greater than a threshold value, it is preferable that thesystem control unit 11 stops continuous shooting and cools the imagingelement 3. In this case, it is possible to prevent the deterioration ofimage quality.

The infrared imaging device illustrated in FIG. 1 comprises thediaphragm 2. Therefore, the infrared rays which are incident on theimaging element 3 include infrared rays radiated from a diaphragm bladeof the diaphragm 2.

In the above description, the FPN calculation unit 19 calculates FPN,without considering the infrared rays radiated from the diaphragm blade.Next, a method for calculating FPN, considering the infrared raysradiated from the diaphragm blade, in order to further improve theaccuracy of calculating FPN will be described.

FIGS. 5A, 5B, and 5C are diagrams illustrating the diaphragm 2 as viewedfrom the imaging element 3. FIG. 5A illustrates the state of thediaphragm 2 when the F-number=F1 is established, FIG. 5B illustrates thestate of the diaphragm 2 when the F-number=F1.4 is established, and FIG.5C illustrates the state of the diaphragm 2 in a state in which thediaphragm 2 is fully closed (F-number=F∞). In FIGS. 5A, 5B, and 5C,reference numeral 2 a indicates an overlap portion between the imagingrange of the imaging element 3 and the outer circumference of thediaphragm blade, reference numeral 2 b indicates the surface of thediaphragm blade, and reference numeral 2 c indicates the opening of thediaphragm 2.

As illustrated in FIGS. 5A, 5B, and 5C in which the diaphragm 2 isviewed from the imaging element 3, when the F-number of the diaphragm 2varies, the exposure area of the diaphragm blade varies. For example,the comparison between the F-number=F∞ and the F-number=F1.4 shows thatthe exposure area of the diaphragm blade when the F-number=F1.4 isestablished is about half of the exposure area of the diaphragm bladewhen the F-number=F∞ is established. The exposure area of the diaphragmblade means the area of a portion of the diaphragm blade which is notshielded by an obstacle that blocks infrared rays, as viewed from theimaging element 3.

Infrared rays are radiated from an exposed portion of the diaphragmblade and are then incident on the imaging element 3. Therefore, it ispreferable that the ratio of the signal component SG(1) of the firstcaptured image data obtained at an F-number of F1 to the signalcomponent SG(2) of the second captured image data obtained at anF-number of F1.4 is calculated considering the amount of infrared raysradiated from the diaphragm 2.

FIG. 6 is a diagram illustrating the relationship between the F-numberand the amount of infrared rays radiated from the diaphragm 2. In FIG.6, the amount of infrared rays β which are radiated from the diaphragm 2when the F-number=F1 is established is standardized to “0” and theamount of infrared rays which are radiated from the diaphragm 2 when theF-number=F∞ is established is standardized to “100”.

As illustrated in FIG. 6, the amount of infrared rays β which areradiated from the diaphragm 2 when the F-number=F1.4 is established isabout half of the amount of infrared rays which are radiated from thediaphragm 2 when the F-number=F∞ is established. Data illustrated inFIG. 6 varies depending on the temperature of the diaphragm 2.Therefore, the data illustrated in FIG. 6 is stored for each temperatureof the diaphragm 2 in the main memory 16.

The pixel signal values which are output from each infrared detectionpixel of the imaging element 3 are different from each other accordingto the amount of infrared rays radiated from the diaphragm 2 and varydepending on a position on a light receiving surface of the infrareddetection pixel.

For example, a pixel signal value which is output from an infrareddetection pixel at a position opposite to an opening portion of thediaphragm 2 is small and a pixel signal value which is output from aninfrared detection pixel at a position opposite to the diaphragm bladeof the diaphragm 2 is large.

A table data in which a pixel signal value that is output from eachinfrared detection pixel according to the infrared rays radiated fromthe diaphragm 2 is recorded for each settable F-number is stored in themain memory 16 of the infrared imaging device illustrated in FIG. 1 inadvance.

In addition, a plurality of table data items corresponding to anarbitrary F-number are stored for each temperature of the diaphragm 2.The reason is that, when the temperature of the diaphragm 2 varies, theamount of infrared rays radiated from the diaphragm 2 varies even if theexposure area of the diaphragm blade is the same.

For example, a signal value corresponding to the temperature of theobject of which the temperature is known and which has a uniformtemperature distribution is subtracted from each pixel signal value ofthe captured image data obtained by capturing the image of the objectusing the imaging element 3 to calculate a pixel signal valuecorresponding to the infrared rays radiated from the diaphragm 2 foreach infrared detection pixel.

This operation can be repeated while the F-number is changed tocalculate the table data. In addition, the operation can be repeated ata fixed F-number while the temperature of the diaphragm 2 is changed togenerate a plurality of table data items for the same F-number.

The pixel signal value that is output from each infrared detection pixelaccording to the infrared rays radiated from the diaphragm 2 isproportional to the product of the fourth power of the temperature ofthe diaphragm 2 and the exposure area of the diaphragm 2.

Therefore, the pixel signal value that is output from each infrareddetection pixel according to the infrared rays radiated from thediaphragm 2 can be calculated for an arbitrary F-number (a value otherthan the minimum value) and can be corrected on the basis of thetemperature and F-number of the diaphragm 2 to create the table data.

FIG. 7 is a diagram illustrating a modification example of the FPNcalculation method.

FIG. 7 illustrates a pixel signal value 41 at an arbitrary coordinateposition of the first captured image data obtained in a state in whichthe F-number=F1 is established and a pixel signal value 42 at thearbitrary coordinate position of the second captured image data obtainedin a state in which the F-number=F1.4 is established.

A signal component SG included in each of the pixel signal value 41 andthe pixel signal value 42 includes a signal component SG(obj)corresponding to infrared rays radiated from the object and a signalcomponent SG(ap) corresponding to infrared rays radiated from thediaphragm 2. In a case in which the F-number=F1 is established, theexposure area of the diaphragm 2 is “0”. Therefore, the signal componentSG(ap) of the pixel signal value 41 is“0”.

The ratio of the signal component SG(obj) of the pixel signal value 42to the signal component SG(obj) of the pixel signal value 41 isdetermined by the ratio of the infrared transmittance α at F1 to theinfrared transmittance α at F1.4 and is about “0.5”. In addition, thesignal component SG(ap) of the pixel signal value 41 and the signalcomponent SG(ap) of the pixel signal value 42 are known values in thetable data.

The FPN calculation unit 19 searches for the signal component SG(ap) ofthe pixel signal value 41 from the table data corresponding to acombination of the temperature of the diaphragm 2 detected by thetemperature detection unit 4 and the F-number=F1 and acquires the signalcomponent SG(ap).

In addition, the FPN calculation unit 19 searches for the signalcomponent SG(ap) of the pixel signal value 42 from the table datacorresponding to a combination of the temperature of the diaphragm 2detected by the temperature detection unit 4 and the F-number=F1.4 andacquires the signal component SG(ap).

Then, the FPN calculation unit 19 sets a coefficient for making a pixelsignal value 41 a obtained by subtracting the acquired signal componentSG(ap) from the pixel signal value 41 equal to a pixel signal value 42 aobtained by subtracting the acquired signal component SG(ap) from thepixel signal value 42.

Then, the FPN calculation unit 19 multiplies at least one of the pixelsignal value 41 a or the pixel signal value 42 a by the set coefficient.A process after the multiplication is the same as that described in FIG.2.

As described above, FPN is calculated, considering a change in theamount of infrared rays radiated from the diaphragm 2 due to a change inthe F-number. Therefore, it is possible to calculate FPN with highaccuracy and to improve image quality.

In this embodiment, detection sensitivity indicating a pixel signalvalue which is output from the infrared detection pixel with respect tothe same amount of infrared rays is the same at any position on thelight receiving surface of the imaging element 3. However, in practice,the detection sensitivity of the infrared detection pixel variesdepending on a position on the light receiving surface of the infrareddetection pixel and shading occurs in the captured image data.

FIG. 8 is a diagram illustrating the detection sensitivity γ of eachinfrared detection pixel when the imaging element 3 captures an image ofan object with a uniform temperature. In FIG. 8, the horizontal axisindicates a position on the light receiving surface in one direction anda position where the horizontal axis intersects the optical axis of theimaging optical system is represented by 0.

“R” indicates the position of an infrared detection pixel located at oneend among the infrared detection pixels on a straight line that is seton the light receiving surface of the imaging element 3 and passesthrough an infrared detection pixel at the position “0”. “L” indicatesthe position of an infrared detection pixel located at the other endamong the infrared detection pixels on the straight line.

In FIG. 8, the detection sensitivity γ when the F-number=F1 isestablished is represented by a dashed line and the detectionsensitivity γ when the F-number=F1.4 is established is represented by asolid line. In FIG. 8, the detection sensitivity γ of the infrareddetection pixel at the position “0” is standardized to “100%”.

As illustrated in FIG. 8, the detection sensitivity γ of the infrareddetection pixel is the highest at the center of the light receivingsurface and is reduced toward the end of the light receiving surface.The rate of decrease in the detection sensitivity γ varies depending onthe F-number.

Therefore, the relationship illustrated in FIG. 2 is established for apixel signal value which is output from the infrared detection pixel atthe center of the light receiving surface. However, the relationshipillustrated in FIG. 2 is not established for a pixel signal value whichis output from the infrared detection pixel in an edge part of the lightreceiving surface.

FIG. 9 is a diagram illustrating a pixel signal value which is outputfrom the infrared detection pixel at the position “L” or “R”.

In FIG. 9, reference numeral 31 a indicates a pixel signal valueobtained by capturing an image in a state in which the F-number=F1 isestablished. Reference numeral 32 a indicates a pixel signal valueobtained by capturing an image in a state in which the F-number=F1.4 isestablished. In FIG. 9, the pixel signal values 31 and 32 illustrated inFIG. 2 are represented by dashed lines.

From the relationship illustrated in FIG. 2, a signal component SG ofthe pixel signal value 32 a is less than a signal component SG of thepixel signal value 31 a. However, from the relationship illustrated inFIG. 8, a reduction in the signal component is less than that in theinfrared detection pixel at the center of the light receiving surface.

Therefore, it is preferable to calculate FPN, considering a variation inthe ratio of the signal component SG of the pixel signal value of thefirst captured image data and the signal component SG of the pixelsignal value of the second captured image data caused by a change in theposition on the light receiving surface.

The relationship illustrated in FIG. 2, the relationship illustrated inFIG. 6, and the relationship illustrated in FIG. 8 are known. Therefore,the FPN calculation unit 19 sets a coefficient for making the signalcomponent SG of the pixel signal value of the first captured image dataequal to the signal component SG of the pixel signal value of the secondcaptured image data to a value corresponding to the position of theinfrared detection pixel on the light receiving surface, with referenceto the data illustrated in FIGS. 2, 6, and 8.

The FPN calculation unit 19 multiplies at least one of the signalcomponent SG of the pixel signal value of the first captured image dataor the signal component SG of the pixel signal value of the secondcaptured image data by the set coefficient.

The FPN calculation unit 19 calculates an FPN component C_(fpn) from thedifference between the signal component SG of the pixel signal value ofthe first captured image data and the signal component SG of the pixelsignal value of the second captured image data after the multiplication.

In this way, it is possible to calculate FPN with high accuracy and toimprove image quality.

The above-mentioned FPN calculation method will be described in detailusing numerical expressions.

It is assumed that the digital value of a component corresponding to theamount of infrared rays radiated from the object among the pixel signalvalues at the coordinates (x, y) in the captured image data isS_(obj)(x, y).

It is assumed that the digital value of an FPN component among the pixelsignal values at the coordinates (x, y) in the captured image data isC_(fpn)(x, y).

It is assumed that the infrared transmittance α illustrated in FIG. 2 atan arbitrary F-number is α(F). Here, α(F) is data stored in the mainmemory 16.

It is assumed that a pixel signal value corresponding to the amount ofinfrared rays radiated from the diaphragm 2 when the F-number is anarbitrary value and the temperature of the diaphragm 2 is T among thepixel signal values at the coordinates (x, y) in the captured image datais β(x, y, F, T). Here, β(x, y, F, T) is data which is experimentallycalculated in advance and is recorded in the main memory 16.

It is assumed that the detection sensitivity γ of the infrared detectionpixel at the coordinates (x, y) on the light receiving surface in astate in which the F-number is an arbitrary value is γ(x, y, F). Here,γ(x, y, F) is data which is experimentally calculated in advance and isstored in the main memory 16.

It is assumed that the digital value of the pixel signal value at thecoordinates (x, y) in the captured image data is Out(x, y).

When the above-mentioned data is set in this way, Out(x, y) isrepresented by the following Expression (1).

Out(x,y)=S _(obi)(x,y)×α(F)×γ(x,y,F)+β(x,y,F,T)+C _(fpn)(x,y)  (1)

Therefore, when the F-number=F1 is established, Out(x, y)_(F1) isrepresented by the following Expression (2). When the F-number=F1.4 isestablished, Out(x, y)_(F1.4) is represented by the following Expression(3).

Out(x,y)_(F1) =S _(obi)(x,y)×α(F1)×γ(x,y,F1)+β(x,y,F1,T)+C_(fpn)(x,y)  (2)

Out(x,y)_(F1.4) =S _(obi)(x,y)×α(F1.4)×γ(x,y,F1.4)+β(x,y,F1.4,T)+C_(fpn)(x,y)  (3)

When both sides are divided by σc(F1)×γ(x, y, F1) in Expression (2),Expression (2) can be changed to the following Expression (4).

$\begin{matrix}{{\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}{{Out}\left( {x,y} \right)}_{F\; 1}} = {{S_{obi}\left( {x,y} \right)} + {\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}{\beta \left( {x,y,{F\; 1},T} \right)}} + {\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}{C_{fpn}\left( {x,y} \right)}}}} & (4)\end{matrix}$

When both sides are divided by α(F1.4)×γ(x, y, F1.4) in Expression (3),Expression (3) can be changed to the following Expression (5).

$\begin{matrix}{{\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}{{Out}\left( {x,y} \right)}_{F\; 1.4}} = {{S_{obi}\left( {x,y} \right)} + {\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}{\beta \left( {x,y,{F\; 1.4},T} \right)}} + {\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}{C_{fpn}\left( {x,y} \right)}}}} & (5)\end{matrix}$

When Expression (4) is subtracted from Expression (5), Expression (6) isobtained.

$\begin{matrix}{{{\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}{{Out}\left( {x,y} \right)}_{F\; 1.4}} - {\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}{{Out}\left( {x,y} \right)}_{F\; 1}}} = {{\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}{\beta \left( {x,y,{F\; 1.4},T} \right)}} - {\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}{\beta \left( {x,y,{F\; 1},T} \right)}} + {\left( {\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}} - \frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}} \right){C_{fpn}\left( {x,y} \right)}}}} & (6)\end{matrix}$

When Expression (6) is solved for C_(fpn)(x, y), Expression (7) isobtained.

$\begin{matrix}{{C_{fpn}\left( {x,y} \right)} = \frac{\begin{matrix}{{\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}}\left( {{{Out}\left( {x,y} \right)}_{F\; 1.4} - {\beta \left( {x,y,{F\; 1.4},T} \right)}} \right)} -} \\{\frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}\left( {{{Out}\left( {x,y} \right)}_{F\; 1} - {\beta \left( {x,y,{F\; 1},T} \right)}} \right)}\end{matrix}}{\left( {\frac{1}{{\alpha \left( {F\; 1.4} \right)} \times {\gamma \left( {x,y,{F\; 1.4}} \right)}} - \frac{1}{{\alpha \left( {F\; 1} \right)} \times {\gamma \left( {x,y,{F\; 1}} \right)}}} \right)}} & (7)\end{matrix}$

Since all of the numerical values in Expression (7) are known, it ispossible to calculate the FPN component C_(fpn)(x, y).

In Expression (7), {1/α(F1.4)×γ(x, y, F1.4)} and {1/α(F1)×γ(x, y, F1)}are coefficients which correspond to the difference between theF-number=F1 and the F-number=F1.4 and are used to make the valuesS_(obj)(x, y) included in two pixel signal values, which are acquiredwhile the F-number is changed, equal to each other.

A process in a case in which γ(x, y, F) is “1” and β(x, y, F, T) is “0”in Expression (7) is the same as that described in FIG. 2. A process ina case in which γ(x, y, F) is “1” in Expression (7) is the same as thatdescribed in FIG. 7.

The FPN calculation method when a motion picture is captured has beendescribed above. Next, an operation in a still image capture mode inwhich the imaging element 3 captures an image of an object one time inresponse to, for example, an instruction from the user and recordingdata obtained by processing the captured image data is recorded in therecording medium 21 will be described.

In the still image capture mode, the digital signal processing unit 17combines a plurality of captured image data items acquired while theF-number of the diaphragm 2 is changed to generate one captured imagedata item, generates recording data from the captured image data, andrecords the recording data in the recording medium 21.

FIG. 10 is a diagram illustrating a change in the pixel signal valueoutput from each infrared detection pixel of the imaging element 3 withrespect to the temperature of the object (the output responsecharacteristics of the infrared detection pixel).

In FIG. 10, reference numeral 101 indicates the output responsecharacteristics of each infrared detection pixel in a state in which theF-number=F1 is established. Reference numeral 102 indicates the outputresponse characteristics of each infrared detection pixel in a state inwhich the F-number=F1.4 is established.

Each of the output response characteristics 101 and the output responsecharacteristics 102 indicates a pixel signal value from which the signalcomponent (β (x, y, F, T)) corresponding to the infrared rays radiatedfrom the diaphragm 2 has been excluded.

In a state in which the F-number=F1 is established, a large amount ofinfrared rays radiated from the object is incident on the imagingelement 3. Therefore, the pixel signal value changes rapidly withrespect to the temperature of the object.

In the example illustrated in FIG. 10, in a state in which theF-number=F1 is established, the pixel signal value is saturated for anobject with a temperature of 20° C. or more. Therefore, it is possibleto capture an image of an object with a temperature of less than 20° C.in a state in which the signal-to-noise ratio (S/N ratio) is high.However, it is difficult to accurately capture an image of an objectwith a temperature of 20° C. or more.

In a state in which the F-number=F1.4 is established, the amount ofinfrared rays incident on the imaging element 3 is less than that whenthe F-number=F1 is established. Therefore, the pixel signal valuechanges rapidly with respect to the temperature of the object.

In the example illustrated in FIG. 10, in a state in which theF-number=F1.4 is established, the pixel signal value is not saturatedfor an object with a temperature of less than 80° C. That is, it ispossible to accurately capture an image of the object with a temperatureof less than 80°. However, for the object with a temperature of lessthan 20° C., the pixel signal value is small and the S/N ratio isreduced.

In FIG. 10, the temperature (=20° C.) at which the infrared detectionpixel reaches a saturation level when the F-number=F1 is established isreferred to as a saturation temperature. In addition, the temperature(=80° C.) at which the infrared detection pixel reaches the saturationlevel when the F-number=F1.4 is established is referred to as thesaturation temperature.

As such, when the F-number of the diaphragm 2 varies, the upper limittemperature (saturation temperature) of the object of which the imagecan be accurately captured varies.

It is assumed that a pixel signal value at the coordinate position (x,y) obtained in a state in which the F-number=F1 is established is Out(x,y, F1) and a pixel signal value at the coordinate position (x, y)obtained in a state in which the F-number=F1.4 is established is Out(x,y, F2). The digital signal processing unit 17 performs calculationrepresented by on the following Expression (8) on the basis of thesettings to generate a pixel signal value D(x, y).

D(x,y)={Out(x,y,F1)−β(x,y,F1,T)}×a+{Out(x,y,F1.4)−β(x,y,F1.4,T)}×b  (8)

FIG. 11 is a diagram illustrating an example of a calculationcoefficient a in Expression (8). As illustrated in FIG. 11, thecalculation coefficient a is inversely proportional to the outputresponse characteristics 101 in the range in which the temperature ofthe object is less than 20° C. The calculation coefficient a is “0”,regardless of the temperature of the object, in the range in which thetemperature of the object is equal to or greater than 20° C. When thetemperature of the object is 0° C., the calculation coefficient a is“0.25” which is the ratio of the pixel signal value of the outputresponse characteristics 101 to the pixel signal value of the outputresponse characteristics 102 when the temperature of the object is 20°C. in FIG. 10.

FIG. 12 is a diagram illustrating an example of a calculationcoefficient b in Expression (8). As illustrated in FIG. 12, thecalculation coefficient b is proportional to the output responsecharacteristics 101 in the range in which the temperature of the objectis less than 20° C. The calculation coefficient b is “1”, regardless ofthe temperature of the object, in the range in which the temperature ofthe object is equal to or greater than 20° C. When the temperature ofthe object is 0° C., the calculation coefficient b is “0”.

The calculation coefficient a corresponds to the output responsecharacteristics 101 (information indicating the relationship between thetemperature of the object and the pixel signal value output from eachinfrared detection pixel of the imaging element) when the F-number=F1 isestablished.

The calculation coefficient a corresponds to the output responsecharacteristics 102 (information indicating the relationship between thetemperature of the object and the pixel signal value output from eachinfrared detection pixel of the imaging element) when the F-number=F1.4is established.

The captured image data obtained by calculation represented byExpression (8) is equivalent to image data which is obtained bycapturing an image of the object using an infrared detection pixelhaving output response characteristics represented by reference numeral130 in FIG. 13.

The output response characteristics represented by the reference numeral130 are the same as the output response characteristics 102 illustratedin FIG. 10. However, the result of the calculation represented byExpression (8) proves that, as the temperature of the object is reducedin the range in which the temperature of the object is equal to or lessthan 20° C., the proportion of {Out(x, y, F1)−β(x, y, F1, T)} to thepixel signal value D(x, y) after composition increases.

In {Out(x, y, F1)−β(x, y, F1, T)}, the number of signal components isrelatively high and the S/N ratio is higher than that in {Out(x, y,F1.4)−β(x, y, F1.4, T)}. Therefore, when an image of an object iscaptured by the infrared detection pixel with the output responsecharacteristics represented by reference numeral 130 in FIG. 13, it ispossible to capture an image of an object with a low temperature of lessthan 20° C. in a state in which each pixel signal value is not saturatedand the S/N ratio is higher than that when the F-number=F1.4 isestablished.

In addition, it is possible to capture an image of an object in atemperature range that is equal to or greater than 20° C. and less than80° C., without saturating each pixel signal value. Therefore, it ispossible to improve imaging quality even in a scene in which both alow-temperature object and a high-temperature object are present.

In the above description, a processing of weighting a plurality of pixelsignal values, which are acquired while the F-number is changed, withthe calculation coefficients a and b and adding the pixel signal valuesis performed for each infrared detection pixel of the imaging element 3to generate one captured image data item.

As a modification example, the digital signal processing unit 17subtracts data corresponding to the infrared rays radiated from thediaphragm 2 from each of a plurality of captured image data items whichare acquired while the F-number is changed. The digital signalprocessing unit 17 cuts out unsaturated portions from a plurality ofcaptured image data items obtained by the subtraction and connects theout-out portions to generate one captured image data item.

FIG. 14 is a diagram illustrating an example of the object of which theimage is captured by the infrared imaging device illustrated in FIG. 1.FIG. 14 illustrates objects including objects 140, 141, and 142. In FIG.14, it is assumed that the object 140 and the object 141 have the sametemperature and the temperature of the object 142 is higher than thetemperature of the objects 140 and 141.

FIGS. 15A and 15B are diagrams illustrating captured image data which isobtained by capturing the image of the objects illustrated in FIG. 14while the F-number of the diaphragm 2 is changed to two values.

FIG. 15A illustrates captured image data 151 which is captured at anF-number of, for example, F1. FIG. 15B illustrates captured image data152 which is captured at an F-number of, for example, F1.4.

In FIGS. 15A and 15B, reference numeral 142 a indicates datacorresponding to the object 142 in FIG. 14. Reference numeral 140 aindicates data corresponding to the object 140 in FIG. 14. Referencenumeral 141 a indicates data corresponding to the object 141 in FIG. 14.

In the examples illustrated in FIGS. 15A and 15B, each pixel signalvalue of the image data portion 142 a reaches the saturation level at anF-number of F1 and each pixel signal value of the image data portion 142a is less than the saturation level at an F-number of F1.4.

At an F-number of F1 and an F-number of F1.4, each of the pixel signalvalues of the image data portions 140 a and 141 a is less than thesaturation level. The level of each pixel signal value of the image dataportions 140 a and 141 a at an F-number of F1.4 is lower than the levelof each pixel signal value of the image data portions 140 a and 141 a atan F-number of F1.

In the examples illustrated in FIGS. 15A and 15B, the digital signalprocessing unit 17 detects the image data portion 142 a in which a pixelsignal value reaches the saturation level from the captured image data151 and cuts out a portion other than the image data portion 142 a fromthe captured image data 151.

The digital signal processing unit 17 subtracts the signal value (β(x,y, F1, T)) corresponding to the amount of infrared rays, which areradiated from the diaphragm 2 and are based on the F-number when thecaptured image data 151 is acquired and the temperature of the diaphragm2 detected by the temperature detection unit 4, from each pixel signalvalue of the cut-out portion to generate first trimming data.

Then, the digital signal processing unit 17 compares each pixel signalvalue of an image data portion corresponding to the detected image dataportion in the captured image data 152 with the saturation level todetermine whether each pixel signal value is less than the saturationlevel. When each pixel signal value is less than the saturation level,the digital signal processing unit 17 cuts out the image data portionfrom the captured image data 152.

The digital signal processing unit 17 subtracts the signal value (β(x,y, F1.4, T)) corresponding to the amount of infrared rays, which areradiated from the diaphragm 2 and are based on the F-number when thecaptured image data 152 is captured and the temperature of the diaphragm2 detected by the temperature detection unit 4, from each pixel signalvalue of the cut-out portion to generate second trimming data.

Then, the digital signal processing unit 17 connects the first trimmingdata and the second trimming data to combine two captured image dataitems 151 and 152 and generates composite image data 155 illustrated inFIG. 16. The digital signal processing unit 17 processes the compositeimage data 155 to obtain recording data.

As described above, in a case in which a plurality of captured imagedata items acquired while the F-number is changed include captured imagedata (referred to as captured image data (1)) including an objectportion in which a pixel signal value reaches the saturation level, thedigital signal processing unit 17 specifies, from the plurality ofcaptured image data items, captured image data which corresponds to theobject portion, includes an object portion having a pixel signal valueless than the saturation level, and is captured at the minimum F-number.

Then, the digital signal processing unit 17 substitutes the objectportion of the captured image data (1) with data of the object portionincluded in the specified captured image data. In this way, even in ascene in which both a low-temperature object and a high-temperatureobject are present, it is possible to accurately capture the images ofthe two objects and to obtain composite image data.

In a case in which the plurality of captured image data items acquiredwhile the F-number is changed do not include the captured image dataincluding the object portion in which the pixel signal value reaches thesaturation level, the digital signal processing unit 17 may processcaptured image data which is captured at the minimum F-number togenerate recording data.

In the examples illustrated in FIGS. 15A and 15B, imaging is performedtwo times while the F-number is changed. However, imaging may beperformed three or more times while the F-number is changed.

For example, the following case is considered: captured image dataobtained at an F-number of F1 includes two saturated object portions andone of the two portions is saturated even at an F-number of F1.4 and isnot saturated at an F-number of F2.

In this case, two object portions of the captured image data obtained atan F-number of F1 may be substituted with an object portion included inthe captured image data obtained at an F-number of F1.4 and an objectportion included in the captured image data obtained at an F-number ofF2.

In the still image capture mode, as described above, imaging isperformed a plurality of times while the F-number is changed. Therefore,it is possible to perform the above-mentioned FPN calculation processusing a plurality of captured image data items obtained by a pluralityof imaging operations. The digital signal processing unit 17 maysubtract FPN data obtained by the FPN calculation process from thecomposite image data to perform FPN correction.

A program that causes a computer to perform each step performed by theFPN calculation unit 19 and the digital signal processing unit 17according to this embodiment may be provided. The program is recorded ina non-transitory computer readable recording medium.

Examples of the “computer readable recording medium” include an opticalmedium, such as a compact disc-ROM (CD-ROM), and a magnetic recordingmedium, such as a memory card. In addition, the program may be providedby downloading through a network.

As described above, the specification discloses the following.

A disclosed infrared imaging device comprises an imaging elementincluding a plurality of infrared detection pixels which aretwo-dimensionally arranged, a diaphragm that is provided closer to anobject than the imaging element, a temperature detection unit thatdetects a temperature of the diaphragm, and an image processing unitthat subtracts, from at least a portion of each of a plurality ofcaptured image data items obtained by capturing images using the imagingelement in a state in which an F-number of the diaphragm is set to aplurality of values, a signal value corresponding to an amount ofinfrared rays which are radiated from the diaphragm and are based on theF-number when each of the plurality of captured image data items isacquired and the temperature detected by the temperature detection unitand combines the plurality of captured image data items after thesubtraction to generate composite image data.

In the disclosed infrared imaging device, in captured image dataincluding an object portion in which a pixel signal value reaches asaturation level among the plurality of captured image data items, theimage processing unit subtracts the signal value from a portion otherthan the object portion to obtain a first portion. In captured imagedata that includes an object portion which corresponds to the objectportion and in which a pixel signal value is less than the saturationlevel, is captured at a minimum F-number, and is specified from theplurality of captured image data items, the image processing unitsubtracts the signal value from the object portion to obtain a secondportion. The image processing unit combines the first portion and thesecond portion to generate the composite image data.

In the disclosed infrared imaging device, the image processing unitsubtracts the signal value from each pixel signal value of each of theplurality of captured image data items, multiplies each pixel signalvalue of each of the plurality of captured image data items after thesubtraction by a coefficient corresponding to information indicating arelationship between the temperature of the object and the pixel signalvalue output from each infrared detection pixel of the imaging element,which is determined by the F-number when each of the plurality ofcaptured image data items is acquired, and adds the pixel signal valuesafter the multiplication in the plurality of captured image data itemsto generate the composite image data.

In the disclosed infrared imaging device, the plurality of capturedimage data items include first captured image data which is obtained bycapturing an image using the imaging element in a state in which theF-number of the diaphragm is set to a first value and second capturedimage data which is obtained by capturing an image using the imagingelement in a state in which the F-number of the diaphragm is set to asecond value. The disclosed infrared imaging device further comprises afixed pattern noise calculation unit that calculates fixed pattern noiseincluded in the captured image data obtained by capturing an image usingthe imaging element, on the basis of the first captured image data, thesecond captured image data, the first value, and the second value

In the disclosed infrared imaging device, the fixed pattern noisecalculation unit multiplies at least one of two pixel signal values atan arbitrary coordinate position in each of the first captured imagedata and the second captured image data by a coefficient correspondingto a difference between the first value and the second value andcalculates the fixed pattern noise at the arbitrary coordinate positionusing a difference between the two pixel signal values after themultiplication.

In the disclosed infrared imaging device, the fixed pattern noisecalculation unit calculates the fixed pattern noise on the basis of thefirst captured image data, the second captured image data, the firstvalue, the second value, and the temperature of the diaphragm detectedby the temperature detection unit.

In the disclosed infrared imaging device, the fixed pattern noisecalculation unit multiplies at least one of two pixel signal values atan arbitrary coordinate position in each of the first captured imagedata and the second captured image data by a coefficient correspondingto a difference between the first value and the second value and thetemperature detected by the temperature detection unit and calculatesthe fixed pattern noise at the arbitrary coordinate position using adifference between the two pixel signal values after the multiplication.

In the disclosed infrared imaging device, the fixed pattern noisecalculation unit uses a value that varies depending on the position ofthe infrared detection pixel as the coefficient.

A disclosed image processing method is performed by an infrared imagingdevice including an imaging element including a plurality of infrareddetection pixels which are two-dimensionally arranged, a diaphragm thatis provided closer to an object than the imaging element, and atemperature detection unit that detects a temperature of the diaphragm.The image processing method comprises an image processing step ofsubtracting, from at least a portion of each of a plurality of capturedimage data items obtained by capturing images using the imaging elementin a state in which an F-number of the diaphragm is set to a pluralityof values, a signal value corresponding to an amount of infrared rayswhich are radiated from the diaphragm and are based on the F-number wheneach of the plurality of captured image data items is acquired and thetemperature detected by the temperature detection unit and combining theplurality of captured image data items after the subtraction to generatecomposite image data.

In the disclosed image processing method performed by the infraredimaging device, in the image processing step, in captured image dataincluding an object portion in which a pixel signal value reaches asaturation level among the plurality of captured image data items, thesignal value is subtracted from a portion other than the object portionto obtain a first portion. In captured image data that includes anobject portion which corresponds to the object portion and in which apixel signal value is less than the saturation level, is captured at aminimum F-number, and is specified from the plurality of captured imagedata items, the signal value is subtracted from the object portion toobtain a second portion. The first portion and the second portion arecombined to generate the composite image data.

In the disclosed image processing method performed by the infraredimaging device, in the image processing step, the signal value issubtracted from each pixel signal value of each of the plurality ofcaptured image data items, each pixel signal value of each of theplurality of captured image data items after the subtraction ismultiplied by a coefficient corresponding to information indicating arelationship between the temperature of the object and the pixel signalvalue output from each infrared detection pixel of the imaging element,which is determined by the F-number when each of the plurality ofcaptured image data items is acquired, and the pixel signal values afterthe multiplication in the plurality of captured image data items areadded to generate the composite image data.

In the disclosed image processing method performed by the infraredimaging device, the plurality of captured image data items include firstcaptured image data which is obtained by capturing an image using theimaging element in a state in which the F-number of the diaphragm is setto a first value and second captured image data which is obtained bycapturing an image using the imaging element in a state in which theF-number of the diaphragm is set to a second value. The image processingmethod further comprises a fixed pattern noise calculation step ofcalculating fixed pattern noise included in the captured image dataobtained by capturing an image using the imaging element on the basis ofthe first captured image data, the second captured image data, the firstvalue, and the second value.

In the disclosed image processing method performed by the infraredimaging device, in the fixed pattern noise calculation step, at leastone of two pixel signal values at an arbitrary coordinate position ineach of the first captured image data and the second captured image datais multiplied by a coefficient corresponding to a difference between thefirst value and the second value and the fixed pattern noise at thearbitrary coordinate position is calculated using a difference betweenthe two pixel signal values after the multiplication.

In the disclosed image processing method performed by the infraredimaging device, in the fixed pattern noise calculation step, the fixedpattern noise is calculated on the basis of the first captured imagedata, the second captured image data, the first value, the second value,and the temperature of the diaphragm detected by the temperaturedetection unit.

In the disclosed image processing method performed by the infraredimaging device, in the fixed pattern noise calculation step, at leastone of two pixel signal values at an arbitrary coordinate position ineach of the first captured image data and the second captured image datais multiplied by a coefficient corresponding to a difference between thefirst value and the second value and the temperature detected by thetemperature detection unit and the fixed pattern noise at the arbitrarycoordinate position is calculated using a difference between the twopixel signal values after the multiplication.

In the disclosed image processing method performed by the infraredimaging device, in the fixed pattern noise calculation step, a valuethat varies depending on the position of the infrared detection pixel isused as the coefficient.

A disclosed image processing program causes an infrared imaging deviceincluding an imaging element including a plurality of infrared detectionpixels which are two-dimensionally arranged, a diaphragm that isprovided closer to an object than the imaging element, and a temperaturedetection unit that detects a temperature of the diaphragm to perform animage processing step of subtracting, from at least a portion of each ofa plurality of captured image data items obtained by capturing imagesusing the imaging element in a state in which an F-number of thediaphragm is set to a plurality of values, a signal value correspondingto an amount of infrared rays which are radiated from the diaphragm andare based on the F-number when each of the plurality of captured imagedata items is acquired and the temperature detected by the temperaturedetection unit and combining the plurality of captured image data itemsafter the subtraction to generate composite image data.

INDUSTRIAL APPLICABILITY

The invention is particularly applied to, for example, in-vehiclecameras and is convenient and effective.

The invention has been described in detail above with reference to aspecific embodiment. However, it will be understood by those skilled inthe art that various modifications and changes of the invention can bemade without departing from the scope and spirit of the invention.

This application is based on JP2014-200076 filed Sep. 30, 2014, thecontent of which is incorporated herein by reference.

EXPLANATION OF REFERENCES

-   -   2: diaphragm    -   3: imaging element    -   4: temperature detection unit    -   11: system control unit    -   17: digital signal processing unit (image processing unit)    -   19: FPN calculation unit

What is claimed is:
 1. An infrared imaging device comprising: an imagingelement comprising a plurality of infrared detection pixels which aretwo-dimensionally arranged; a diaphragm that is provided closer to anobject than the imaging element; a temperature detection unit thatdetects a temperature of the diaphragm; and an image processing unitthat subtracts, from at least a portion of each of a plurality ofcaptured image data items obtained by capturing images with the imagingelement in a state in which an F-number of the diaphragm is set to aplurality of values, a signal value corresponding to an amount ofinfrared rays which are radiated from the diaphragm and are based on theF-number when each of the plurality of captured image data items isacquired and the temperature detected by the temperature detection unit,and combines the plurality of captured image data items after thesubtraction to generate composite image data.
 2. The infrared imagingdevice according to claim 1, wherein, in captured image data includingan object portion in which a pixel signal value reaches a saturationlevel among the plurality of captured image data items, the imageprocessing unit subtracts the signal value from a portion other than theobject portion to obtain a first portion, in captured image data thatincludes an object portion which corresponds to the object portion andin which a pixel signal value is less than the saturation level, iscaptured at a minimum F-number, and is specified from the plurality ofcaptured image data items, the image processing unit subtracts thesignal value from the object portion to obtain a second portion, and theimage processing unit combines the first portion and the second portionto generate the composite image data.
 3. The infrared imaging deviceaccording to claim 1, wherein the image processing unit subtracts thesignal value from each pixel signal value of each of the plurality ofcaptured image data items, multiplies each pixel signal value of each ofthe plurality of captured image data items after the subtraction by acoefficient corresponding to information indicating a relationshipbetween the temperature of the object and the pixel signal value outputfrom each infrared detection pixel of the imaging element, which isdetermined by the F-number when each of the plurality of captured imagedata items is acquired, and adds the pixel signal values after themultiplication in the plurality of captured image data items to generatethe composite image data.
 4. The infrared imaging device according toclaim 1, further comprising: a fixed pattern noise calculation unit,wherein the plurality of captured image data items include firstcaptured image data which is obtained by capturing an image with theimaging element in a state in which the F-number of the diaphragm is setto a first value and second captured image data which is obtained bycapturing an image with the imaging element in a state in which theF-number of the diaphragm is set to a second value, and the fixedpattern noise calculation unit calculates fixed pattern noise includedin the captured image data obtained by capturing an image with theimaging element, based on the first captured image data, the secondcaptured image data, the first value, and the second value.
 5. Theinfrared imaging device according to claim 4, wherein the fixed patternnoise calculation unit multiplies at least one of two pixel signalvalues at an arbitrary coordinate position in each of the first capturedimage data and the second captured image data by a coefficientcorresponding to a difference between the first value and the secondvalue, and calculates the fixed pattern noise at the arbitrarycoordinate position using a difference between the two pixel signalvalues after the multiplication.
 6. The infrared imaging deviceaccording to claim 4, wherein the fixed pattern noise calculation unitcalculates the fixed pattern noise based on the first captured imagedata, the second captured image data, the first value, the second value,and the temperature of the diaphragm detected by the temperaturedetection unit.
 7. The infrared imaging device according to claim 6,wherein the fixed pattern noise calculation unit multiplies at least oneof two pixel signal values at an arbitrary coordinate position in eachof the first captured image data and the second captured image data by acoefficient corresponding to a difference between the first value andthe second value and the temperature detected by the temperaturedetection unit, and calculates the fixed pattern noise at the arbitrarycoordinate position using a difference between the two pixel signalvalues after the multiplication.
 8. The infrared imaging deviceaccording to claim 5, wherein the fixed pattern noise calculation unituses a value that varies depending on the position of the infrareddetection pixel as the coefficient.
 9. An image processing method thatis performed by an infrared imaging device including an imaging elementincluding a plurality of infrared detection pixels which aretwo-dimensionally arranged, a diaphragm that is provided closer to anobject than the imaging element, and a temperature detection unit thatdetects a temperature of the diaphragm, the method comprising: an imageprocessing step of subtracting, from at least a portion of each of aplurality of captured image data items obtained by capturing images withthe imaging element in a state in which an F-number of the diaphragm isset to a plurality of values, a signal value corresponding to an amountof infrared rays which are radiated from the diaphragm and are based onthe F-number when each of the plurality of captured image data items isacquired and the temperature detected by the temperature detection unitand combining the plurality of captured image data items after thesubtraction to generate composite image data.
 10. The image processingmethod according to claim 9, wherein, in the image processing step, incaptured image data including an object portion in which a pixel signalvalue reaches a saturation level among the plurality of captured imagedata items, the signal value is subtracted from a portion other than theobject portion to obtain a first portion, in captured image data thatincludes an object portion which corresponds to the object portion andin which a pixel signal value is less than the saturation level, iscaptured at a minimum F-number, and is specified from the plurality ofcaptured image data items, the signal value is subtracted from theobject portion to obtain a second portion, and the first portion and thesecond portion are combined to generate the composite image data. 11.The image processing method according to claim 9, wherein, in the imageprocessing step, the signal value is subtracted from each pixel signalvalue of each of the plurality of captured image data items, each pixelsignal value of each of the plurality of captured image data items afterthe subtraction is multiplied by a coefficient corresponding toinformation indicating a relationship between the temperature of theobject and the pixel signal value output from each infrared detectionpixel of the imaging element, which is determined by the F-number wheneach of the plurality of captured image data items is acquired, and thepixel signal values after the multiplication in the plurality ofcaptured image data items are added to generate the composite imagedata.
 12. The image processing method according to claim 9, furthercomprising: a fixed pattern noise calculation step, wherein theplurality of captured image data items include first captured image datawhich is obtained by capturing an image with the imaging element in astate in which the F-number of the diaphragm is set to a first value andsecond captured image data which is obtained by capturing an image withthe imaging element in a state in which the F-number of the diaphragm isset to a second value, and in the fixed pattern noise calculation step,fixed pattern noise included in the captured image data obtained bycapturing an image with the imaging element is calculated based on thefirst captured image data, the second captured image data, the firstvalue, and the second value.
 13. The image processing method accordingto claim 12, wherein, in the fixed pattern noise calculation step, atleast one of two pixel signal values at an arbitrary coordinate positionin each of the first captured image data and the second captured imagedata is multiplied by a coefficient corresponding to a differencebetween the first value and the second value and the fixed pattern noiseat the arbitrary coordinate position is calculated using a differencebetween the two pixel signal values after the multiplication.
 14. Theimage processing method according to claim 12, wherein, in the fixedpattern noise calculation step, the fixed pattern noise is calculatedbased on the first captured image data, the second captured image data,the first value, the second value, and the temperature of the diaphragmdetected by the temperature detection unit.
 15. The image processingmethod according to claim 14, wherein, in the fixed pattern noisecalculation step, at least one of two pixel signal values at anarbitrary coordinate position in each of the first captured image dataand the second captured image data is multiplied by a coefficientcorresponding to a difference between the first value and the secondvalue and the temperature detected by the temperature detection unit andthe fixed pattern noise at the arbitrary coordinate position iscalculated using a difference between the two pixel signal values afterthe multiplication.
 16. The image processing method according to claim13, wherein, in the fixed pattern noise calculation step, a value thatvaries depending on the position of the infrared detection pixel is usedas the coefficient.
 17. A computer readable medium storing an imageprocessing program that causes an infrared imaging device including animaging element including a plurality of infrared detection pixels whichare two-dimensionally arranged, a diaphragm that is provided closer toan object than the imaging element, and a temperature detection unitthat detects a temperature of the diaphragm to perform: an imageprocessing step of subtracting, from at least a portion of each of aplurality of captured image data items obtained by capturing images withthe imaging element in a state in which an F-number of the diaphragm isset to a plurality of values, a signal value corresponding to an amountof infrared rays which are radiated from the diaphragm and are based onthe F-number when each of the plurality of captured image data items isacquired and the temperature detected by the temperature detection unitand combining the plurality of captured image data items after thesubtraction to generate composite image data.